Omni-Directional Mobile Robot for Remote Users

ABSTRACT

A robot uses controlled omni-wheels and an arm assembly coupled to a gripper to facilitate the opening of doors. The gripper is rotatable around at least one of its axis to engage and rotate a door handle to unlatch a door. The omni-wheels are driven to move the robot along a curved path to open the door.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of allowed U.S. application Ser. No.16/042,153, filed on Jul. 23, 2018 and entitled “OMNI-DIRECTIONAL MOBILEMANIPULATOR”, which is a continuation of U.S. application Ser. No.15/255,935, filed on Sep. 2, 2016 (now U.S. Pat. No. 10/029,370) andentitled “CONTROL SYSTEM FOR MOBILE ROBOT”, which is a divisional ofU.S. application Ser. No. 14/662,431, filed Mar. 19, 2015 (now U.S. Pat.No. 9,440,356) and entitled “CUSTOMIZABLE SYSTEM FOR MOBILE ROBOT”,which is a continuation of U.S. application Ser. No. 13/806,382, filedDec. 21, 2012 (now U.S. Pat. No. 8,994,776) and entitled “CUSTOMIZABLEROBOTIC SYSTEM”, which is a National Stage Entry of PCT/CA2011/001251,filed on Nov. 14, 2011, which claims priority from Canadian ApplicationNo. 2,720,886 filed on Nov. 12, 2010 all of which are incorporatedherein by reference.

BACKGROUND OF THE INVENTION

The present invention is in the technical field of robotic applicationsand in particular is useful in a mobile platform suitable for variouscommunication systems that include a robotic presence. In a preferredapplication a virtual presence robotic device is provided thatadvantageously uses a number of existing computer arrangements to reducecost and extend the applications and customizations.

A robotic virtual presence provides telepresence functionality thatincludes enabling an experience of presence at a location which isremote from that of the physical being, and also provides those who arein the presence of the robot, the experience that someone is beingprojected through the robot.

Telepresence systems ideally require a communication access devicethrough which bi-directional audio and video is carried between the user(or users) and the remote location of the robot along with motioninstructions to enable the user to freely explore remote surroundings.The telephone could be considered the first generation of telepresencein that it bi-directionally carries only one of mans' senses, that ofhearing, almost instantaneously across virtually any distance spanningtwo physical beings. This one-to-one form of communication, whereby eachparticipant requires an access device conforming to a globalconnectivity standard known as the POTS system, remains ubiquitoustoday.

Telephones are available with a variety of customized features meetingwide ranging communications applications including business, hospitaland personal needs. For example, conference calling capabilities wereinvented which enabled any number of communication access devices, inany number of locations, to tie together to enable multiple parties toconcurrently converse. This many-to-many form of communication iscommonly used throughout the world today as the cost per minute ofcommunications, and the inclusion of standard telephones as compatibleaccess devices, enables the service to be easily afforded and accessedby the population at large.

The next leap in telepresence occurred when real-time, bi-directionaltransmission of a second sense of man, that of sight, between two beingswas commercially deployed by AT&T over 3 decades ago. Known as thePICTUREPHONE™, the one-to-one service was a commercial failure with lowadoption due to high access device costs. Although backward compatiblewith any POTS access device, without a large number of persons equippedto communicate via PICTUREPHONES™, and no method of identifying at theoutset of a call who had PICTUREPHONE™ capabilities, and no options toenhance or specifically target the performance of the PICTUREPHONE™ forvertical markets, it was difficult to justify the $1000-plus cost perdevice.

Cellular and VOIP phones adding mobile telepresence access, whenlaunched, also included features ensuring their compatibility withexisting POTS infrastructure. Today, numerous videoconferencing systemsemploying common display monitors and projectors, speakers, microphones,and cameras spanning inexpensive laptop, game console, and televisionweb-cam linkups through to dedicated multi-media conference rooms existwhich primarily leverage now ubiquitous web connectivity channels, butstill typically include support for standard POTS access to the voicechannel by those without specialized access devices. Many proprietaryvideo extensions also provide services which make it quick and easy tofind which contacts have compatible video capabilities and most advancedsystems support many-to-many communications via sight and sound.

Despite advancements in telepresence systems, each participant muststill pro-actively engage in the establishment of the conference—andactivate, typically via a key press or voice command, an access deviceat the outset of such communication. Even the latest videophoneapplication, Apple's FACETIME™, requires that the launch of thecommunication channel be established by two people who must both be inphysical connection with the access device—in this case, an Apple IPOD™or IPHONE™.

The end objective of robotic telepresence systems is to create a trueremote representation, or presence, of the individual being. But thishas yet to be achieved at a cost acceptable to, or with any catalystfor, mass adoption. Each system proposed offers no globalinter-connectivity with other systems, other than in some cases viabasic POTS. Early robotic-assisted communications systems, dating backover a decade to at least the embodiments disclosed in Canadian PatentNo. 2289697 issued to Treviramus et. al., teach of robotic mechanismsfor moving both the monitor and camera to face sources of sound. Othersystems, such as that disclosed in U.S. Pat. No. 7,171,286 issued toWang, et. al., teach of more elaborate controls and motion mechanismsenabling the control of any linked robot from any other robot tofacilitate both auditory and visual communications as well as a means toprioritize access requests. Another teaching, in U.S. Pat. No. 6,292,713issued to Jouppi, et. al., provides a comprehensive framework for atelepresence device complete with an array of cameras and speakers, yetthe application ignores any system enabling ubiquity, and related costconsiderations.

As a distinct science from that of telepresence, early virtual presenceconcepts could be thought to have been first explored during thedevelopment of a program known as ELIZA™, written at the MassachusettsInstitute of Technology between 1964 and 1966 by J. Weizenbaum whichsimulated the questions and responses of a hypothetical psychotherapistthrough a teletype interface. The author of the present invention alsowrote a similar program with much larger vocabulary in 1974 at UpperCanada College which, also accessed via a teletype, consumed a verysignificant portion of a HEWLETT-PACKARD™ minicomputer's memory duringoperation. Observers in both cases found the illusion that a real personwas responding to their comments and questions entered at the teletypevery compelling despite the lack of auditory or visual clues. Thealgorithms developed at Upper Canada College were also able to userudimentary rules to classify and store input, providing the additionalillusion that the ‘person’ at the remote end of the teletype waslearning from day to day as the vocabulary expanded.

There remains a need for a true telepresence robot which can project theindividual into a remote location without the need for each end of acommunication session to be activated by a participant. There furtherremains a need for a customizable telepresence platform supporting awide breadth of usage scenarios while ensuring interoperability with lowentry point access devices to succeed in the marketplace. Lastly, thereremains the need for a true virtual presence robot which appears to beoperated remotely by a human, yet is actually a simulation of suchtelepresence projected through visual, auditory, and autonomous motionclues.

SUMMARY OF THE INVENTION

The invention pertains to a mobile robot having a body, a support basesupporting the body and having a plurality of omni-directional wheelsdriven to move the mobile robot across a surface. At least one armassembly is coupled to the body at a first end thereof and coupled to agripper at a second end thereof. The gripper is selectively movablebetween an open position and closed positon.

In a preferred embodiment, the gripper is configured to engage a doorhandle coupled to a door and the omni-wheels on the support base aredriven to move the mobile robot along a curved path to open the door.

In a further aspect of the invention, the gripper is rotatable around anaxis of the gripper.

In yet a further aspect of the invention, the gripper is configured toengage and rotate the door handle to unlatch the door and theomni-wheels on the support base are driven to move the mobile robotalong a curved path to open the door.

In a preferred embodiment of the invention, the arm assembly isconfigured to move in the horizontal, vertical planes with the armassembly comprising a plurality of links, each having a longitudinalaxis with the gripper being rotatable about the longitudinal axis of thelink adjacent the gripper.

In a further aspect of the invention, the arm assembly is configured tomove through in the horizontal, vertical planes with the arm assemblycomprising an upper link and lower link, each having a longitudinalaxis. The upper link coupled at a first end to the body of the robotwith a driven first joint and at a second end to a first end of thelower link by a second joint. The second end of the lower link iscoupled to the gripper and the gripper is rotatable about itslongitudinal axis of the lower link.

In yet a further aspect of the invention, the gripper is coupled to thesecond end of the lower link by a driven joint.

In a preferred embodiment, the first joint pivots in one of thehorizontal plane, one vertical plane relative to the support base, thesecond joint pivots on the other horizontal plane and vertical planerelative to the support base, and the third joint rotates about thelongitudinal axis of the gripper.

In another preferred embodiment, the at least one arm assembly comprisesa pair of opposing arm assemblies. Each arm assembly is coupled to thebody at a first end thereof and has a gripper at a second end of the armassembly. The grippers are rotatable around an axis of the gripper andare selectively movable between an open position and closed positon.

In a further aspect of the invention, the support base includes at least3 driven omni-wheels.

The invention further pertains to a method of opening a door comprising:

-   -   driving a plurality of omni-wheels to move a robot to a position        adjacent a door to be opened,    -   using a drive of the robot to move a gripper at the end of an        arm assembly of the robot into an open grip position,    -   positioning the gripper at the door handle,    -   moving the gripper to engage the door handle,    -   driving the omni-wheels to cause movement of the robot along a        curved path to move the door to an open position.

In a preferred method, the step of rotating the gripper to rotate andunlatch the door handle is included.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of the virtual presence robot according toan embodiment of the present invention;

FIG. 2 is a side view of the virtual presence robot according to anembodiment of the present invention;

FIG. 3 is a front view of the virtual presence robot according to anembodiment of the present invention;

FIG. 4 is an external perspective view of the motor base according to anembodiment of the virtual presence robot of the present invention;

FIG. 5 is an internal perspective view of the motor base according to anembodiment of the virtual presence robot of the present invention;

FIG. 6 is an exploded view of the motor configuration within the baseaccording to an embodiment of the virtual presence robot of the presentinvention;

FIG. 7 illustrates the insertion of a commercially available tabletcomputer into the head support according to an embodiment of the virtualpresence robot of the present invention;

FIG. 8 is an alternate head for the virtual presence robot according toan embodiment of the present invention;

FIG. 9 illustrates a sample screen image from a remote host platformjust before a connection to the virtual presence robot is established;

FIG. 10 illustrates a sample screen image from a remote host platformduring connection with the virtual presence robot;

FIG. 11 is a front view of an alternate mid-section module with movablearms;

FIG. 12 is a side view of the alternate mid-section module;

FIG. 13 is a front view of a further alternate mid-section module withtwo cup holder appendages;

FIG. 14 is a side view of the further alternate mid-section module;

FIGS. 15 and 16 are a front view and side view of a modified head modulefor a Microsoft KINECT XBOX 360;

FIG. 17 is an exploded perspective view of a preferred robotic structureshowing the interconnection between the various modules and a mechanicalconnection of a computer display to the head module;

FIG. 18 is a front view of the perspective view of FIG. 17;

FIG. 19 is a side view of the device in FIG. 17;

FIG. 20 is a general schematic showing the communication between thevarious devices; and

FIG. 21 is a schematic of the electrical connection of the variouscomponents.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the invention in more detail, in FIGS. 1 to 3, there isshown an embodiment of the virtual presence robot 2 having four distinctsub-assemblies or modules: the head sub-assembly 12, mid-sectionsub-assembly 14, transition sub-assembly 16 and base sub-assembly 18,which can be quickly and easily assembled with minimal fastening pointsby an unskilled purchaser. The modular design of the virtual presencerobot provides flexibility and allows it to be shipped in a costeffective manner. Sub-assemblies of the illustrated embodiment aredesigned to be inexpensively made of injection-molded plastics, and usetheir exoskeletons as the primary structural members. Each sub-assemblyis also designed to be offered in different functional forms, to meetvarying customer expectations while maintaining compatibility andinteroperability as will be presented in more detail below. Thesub-assemblies may be disassembled as necessary for upgrade orreplacement of any sub-assembly using common connection techniques,making the reconfiguration thereof including remounting and reassemblysimple and fast.

The modular design provides advantages for shipping, however moreimportantly it allows for multiple models and convenient updating orcustomization for particular applications.

Turning specifically to FIG. 1, the head sub-assembly 12 includes one ofa number of possible tilting mounting plates 22 with guide tabs 23designed to securely hold a customer-selected third party mass producedtablet or smartphone. The robot is preferably designed to function withsuch a third party device being properly docked into the mounting plateor otherwise electrically connected via USB. WiFi or cellularconnectivity with remote services and users, and preferably most of theprocessing power necessary to operate the robot, is preferably deliveredthrough the third party head device. A special head sub-assembly thatdoes not require a third party device is shown in FIG. 8.

The driven tilt mechanism 24 enables the third party head device to facedown, straight ahead, or up at varying angles to exhibit emotion, changethe field of view of the camera (assuming the given third party deviceincludes a camera) and in conjunction with the wheels 20 to provideleft/right motion, establishing eye contact between the remote user whois facing out of the screen of the third party head device and theperson or persons in proximity to the virtual presence robot.

The ability to provide eye contact is an effective communicationfunction. In order to enable the head device to convey the remoteoperator's eye contact in the virtual session, two cameras are ideallydeployed at the remote location to capture, and re-center both the headand eye position of the operator. This is done through 3D reconstructionof the operator's head and in particular face, in real time. Facialexpressions are preserved, as is some eye movement. In this way, ifthere are, for example, three people sitting on a couch in front of atelevision connected to the virtual presence robot, algorithms capturingthe scene of the group facing the television will identify the speakerand then take their head and fill the frame of the head device with thisreconstruction, even if the person speaking isn't squarely in front ofthe camera. This ensures that those in the room with the virtualpresence robot feel a stronger connection with the individual speaking.This connection may be further enhanced, should the head device supportthe display of 3D video. Facial reconstruction can also be achieved withone RGB-D camera or even a simple RGB camera and prior knowledge of theparticipant's facial geometry.

The tilt mechanism 24 also compensates for periods when the base 18 isnot level as determined by a gyro in the base—for example, when climbinga wheelchair ramp or in unlikely emergencies if the robot begins to fallover where, in such cases, the tilt mechanism receives commands in anattempt to prevent the third party device from impacting the floor. Insome cases, a more complex tilt mechanism may be deployed to enable thehead to pivot from side to side or extend outwards to project over a bedor wheelchair.

Also shown in FIG. 1 is a large speaker 28 and noise-cancellingmicrophones 30 which are present if the third party device doesn't havesuitable speaker and microphone capabilities for the intended use. Forexample, many third party tablets have suitable speakers and microphonesfor home environments and thus could use a lower-cost head sub-assemblywithout the need for the large speaker. However, for virtual presenceoffice or shopping, the larger speaker and higher quality microphonesmay be preferred such that some third party head devices may have two ormore different head sub-assemblies for users to choose from, dependingupon the breadth of target applications and available budget.

An array of microphones provides the robot with the ability to sense thedirection of sound as well as perform noise-cancelling. Sensing thedirection of sound provides extremely valuable sensory data in numerousscenarios:

-   -   a. security monitoring—enables the robot to sense the direction        of intrusions or other security events (falling trees, water        leakage, window breakage, etc.) to further        investigate—algorithms can use the microphone array to move        towards the sound;    -   b. navigation—when combined with the 360 degree camera, enables        the robot to gracefully move out of the path of people or other        robots moving towards it and also, in the case of guide robotic        uses, enables the robot to guide people (blind and/or hearing        impaired) out of the way of such moving vehicles; and    -   c. health care—enables the robot to hear verbal calls for        assistance and, as desired, move towards such sources of sound        for further investigation; and    -   d. Note that for cost reasons, only one microphone may be used        in some head assemblies where the direction of incoming sound is        not critical.

As will be further discussed, additional microphones and/or speakers maybe provided that are placed within the robot's environment—for examplein a fixed location (perhaps an in-accessible location) to provideadditional sensory data or broaden functionality.

The head sub-assembly 12 also includes status lights and a ‘do notdisturb’ button 26 which when pressed or actuated remotely (from anyauthorized wireless connection), illuminates and prevents externaltelepresence connections with the robot. Assuming the third party headdevice is attached (either physically or wirelessly), it also updatesthe robot's status at the service hub (described in more detail later).Other lights indicate battery charge level and charging state(irrespective of whether the third party head device is connected), whenthe head device is properly seated in the mounting plate 22, or when thehead device is connected to the robot wirelessly via Bluetooth or WiFi,amongst other features.

New head sub-assemblies, and updated applications (where softwareupdates are necessary, such as those typically delivered through thirdparty application stores) are brought to market to correspond withlaunch events for new third party models of smartphones and tablets.Generally, such new consumer electronics devices incorporate increasingprocessing speed and memory, better camera and display technologies, andenhanced wireless communications capabilities. By enabling the virtualpresence robot to support these new hardware advancements via a simpleupgrade to the head sub-assembly, product life is significantlyincreased as the array and quality of capabilities of the robot canimprove over time as existing applications run faster and newapplications become feasible. As previously noted, in some cases, agiven third party device may give rise to two or three different headsub-assemblies ranging in cost based on functionality—the most basic ofwhich would not include the tilt mechanism 24, speaker 28, andmicrophones 30. Elaborate head sub-assemblies could also include cameraarrays (augmenting or replacing the 360 degree imaging apparatuscurrently located in the mid-section 14) where a high vantage point isneeded or to free the mid-section 14, typically housing the 360 degreeimaging apparatus for other uses.

The mid-section 14 is also designed to be available in a variety ofconfigurations. The most basic and inexpensive mid-section has nofunctionality and thus, with this configuration, navigation of thevirtual presence robot must only rely on features of the configured headdevice and the ultrasonic sensors 38 in the base sub-assembly 18.However, this embodiment can successfully employ existing mass marketconsumer electronics devices, such as the Microsoft KINECT, which ifjoined with a tablet or other device in the head sub-assembly, iscapable of providing all necessary sensory data and processing forsuccessful navigation. (For even more inexpensive configurations,Microsoft's KINECT may also be used on its own, as described later andillustrated in FIGS. 15 and 16).

To best support navigation and collision-avoidance with typical currentthird party smartphones and tablet driven heads, mid-sections with atleast a forward down facing camera, or more ideally 360 degree visionencircling the area about the base, is desired. Where more precision orsmoother execution is desired, especially in environments such ashospitals where collisions are absolutely unacceptable yet corridorscongested with other moving traffic are common, or where the motion ofmultiple robots must be orchestrated in addition to the possibility ofincluding remote sensors, a special mid-section could also be customizedto house additional sensors beyond the 360 degree vision sensor,including infrared (consisting of infrared sensor and one or moreinfrared lighting devices), ultrasonic (consisting of separate orcombined emitter and receivers) and laser range finders. In these cases,multiple sensory data streams would be fetched by the head device whichwould perform the processing, optionally while communicating with otherrobots, or the head device could offload some or all of the processingtasks to higher speed remote processing servers which would then sendnear real time direction and speed updates back to the robot. It isfurther advantageous, especially in long run time environments, tooff-load more complex vision and navigation to remote servers which aredirectly connected to the power grid, thereby reducing battery drain andbeing able to leverage the latest CPU and GPU designs. Note that via anAPI, internal processing of the ultrasonic sensor data in the basesib-assembly would be set to only interrupt motion in extremely closeproximity collisions.

For special vertical market uses, such as hospital or nursing home drugdelivery for example, the mid-section module 14 can also be adapted as amedical prescription pill dispensing arrangement with trackingcapability coupled with corresponding software running in the headdevice which would enable access to drugs, and monitoring of theiringestion, at prescribed times each day. The mid-section can alsocontain a tray and optional cup-locking mechanism which can be used todeliver drinks to patients or test samples to a lab (see FIGS. 13 and14).

In conjunction with a remote robotic loading station, the virtualpresence robot can make multiple trips between a loading station andpatients thereby freeing support staff for other more demandingfunctions. At each patient location, by assessing web resources andunder control of a remote service, the robot can, for example, present arealistic rendering of a human face making informative, yet soothing,small-talk of current events like the weather forecast or newshighlights, reading online poems or books, playing music, or commentingon what was going to be fun to watch on TV while waking the patient fora drink and then monitoring the patients ingestion of medications. Suchpatients could additionally have passive identification devices toassist in recognition and delivery. Such vertical applications andrelated mid-section sub-assembly designs could be supplied bythird-parties which would seamlessly integrate into the robot, giventhat the API for the base will support common motor and solenoidcontrol. These third-parties might also develop map libraries andpossibly fit the environment with special ceiling, wall or floor markersto aid in autonomous navigation as directed by their remote service.

There are a number of ways of achieving the desired 360 degree visionincluding multiple camera arrays and fish-eye lenses, any of which mightbe incorporated into various mid-section or head sub-assembly versionsoffered for different configurations, at different price-points, of thevirtual presence robot. The mid-section 14 embodiment in FIG. 1illustrates an inexpensive and effective apparatus generating both a 360degree field of view (or essentially 360 degree field of view) and 3Dvideo using at least two reflective domes 34 facing a video camera 36.According to this embodiment, the 360 degree imaging apparatus has aninexpensive video camera with lens 36 pointing upwards, through the openport 35 in the mid-section, to capture the reflections from the downwardfacing reflective domes 34. Using prior knowledge of both the lens anddome designs, as well as the distance between them, the distortions ofthe reflections captured by the camera 36 are then pre-processed by theembedded microprocessor of the main circuit card 64 (see FIG. 5) in thebase 18 to produce a series of images or video stream, includingoptional distance information in the case of two or more domes, whichcan be effectively used by the autonomous navigation system. Theseimages or the video stream may optionally be further processed,including the creation of 3D video if two or more domes are used, beforedisplay to assist a remote operator with semi-automatic navigation. Tokeep production costs low in the illustrated embodiment, the domes aremanufactured using injection-molded plastic coated with a reflectivesurface. The domes are illustrated as half-spheres, but other shapeswhich provide different distortion patterns, some of which enable a moreeven density of pixels from the camera across the field of the image,can be used.

Any number of domes may be used to create a 360 degree video, although asingle dome 34 as illustrated in FIG. 3 cannot produce a 3D scene. Inthe case of single or twin dome 34 installations, portions of the 360degree field of vision will be obstructed by one or more verticalsupports and in the case of twin domes, a true 3D scene cannot begenerated along the single plane vertically cutting both domes. Giventhat typical motion of the robot is forward or backwards at varyingangles but not often directly sideways, in the embodiment illustrated inFIGS. 1, 2 & 3, the mid-section 14 has a simple, thin profile, yetstructurally sound design which permits two obstructed fields at both 90and 270 degrees from the forward position. As illustrated in FIG. 2, themid-section 14 is generally planar and the domes 34 can be made toextend past the side width of the mid-section. As will be appreciated bythose skilled in the art, the port between the domes and camera inmid-section 14 seen clearly in FIGS. 1 and 3, could be redesigned toremove one of the two supporting sides and by relocating and replacingthe other with a much thinner profile to relocate and reduce the size ofthe obstructed area. However, such redesign would require significantlystronger materials to prevent vibrations from randomly perturbing therelationship between the camera lens and the dome, which is ideallyfixed to ensure consistent post-processing of the scene. A furthermodification to the design could see the entire camera-dome apparatus ona swinging and possibly pivoting horizontal axle such that when therobot is moving up or down wheelchair ramps, the horizon would remainlevel. In the case where a higher view in any direction was desired, themotion of an axle swings the apparatus forward or backward tilting themirrored domes away from the intended point of interest, therebyincreasing the field in that direction.

Where 3 or more domes are installed, a full 360 degree field can beconstructed without any obstructed portions, but assuming a constantcamera 36 pixel count, with each addition of a dome 34, the effectiveresolution of the resulting 360 degree field is significantly reduced,since the camera's field of view must be broadened to encompass alldomes which avail lower pixel coverage for any given dome. An alternatemethod of providing an unobstructed 360 degree field as the robot movesin any direction except exactly in the direction of the obstruction isto interpolate the obstructed pixel data from prior frames inconjunction with data arriving from other sensors including theultrasonic sensors in the base sub-assembly. Such interpolated datacould not effectively represent moving objects such as animals withoutsignificant processing power which may be available in future headdevice upgrades or could be provided by remote servers processing thedata arriving from the robot before re-transmission to the remote user.

The mid-section sub-assembly 14 also includes a laser pointer apparatus32 which in some applications can also function as a laser patternemitter for 3D object reconstruction. As mentioned earlier, an infraredlighting apparatus may also be included in the mid-section sub-assemblywith the appropriate infrared-sensitive camera 36 so that the virtualpresence robot can navigate in relative darkness without disturbingthose sleeping or watching television or playing video games in reducedlight conditions.

A number of transition sub-assemblies 16 can be used to satisfydifferent height requirements as illustrated in FIG. 3 by the arrows 17.The standard transition sub-assembly 16 adds enough height to thevirtual presence robot that the head is at a height comfortable forconversations with people sitting in a living-room or around a kitchentable or lying in bed, yet is not uncomfortable when addressing peoplestanding. It has no electronic or motorized functions, but such may beincluded as desired for specialized applications. For example, in someapplications, this sub-assembly might also contain compartments to holdcups or other objects augmenting certain mid-section functionality,particularly for nursing home and hospital applications, and thesecompartments can include contact or other sensors or latchingmechanisms. The transition sub-assembly may also be provided with anautomated extension capability for specialized applications requiringboth comfortable seating height and standing height head options.

The modularity of the virtual presence robot or mobile platform allowsadaption thereof for use by those confined to wheelchairs or beds. Byaugmenting both the length of the transition sub-assembly and the headsub-assembly to enable articulation by adding two degrees of freedom,the head sub-assembly can then enable the head device to extend eitherover a bed, facing downwards, or across, just above the lap of a personsitting in a wheelchair or sitting upright in a bed. Additional visionsensors that enable image processing algorithms executed in the headdevice to appropriately extend across a wheelchair or over a bed withoutcollision, are relatively inexpensive component add-ons, as the‘infrastructure’ to support these and the additional motors existswithin the virtual presence robot. The present system makes it possibleto thus bring the face of the remote user to confined elderly anddisabled friends and relatives without the need for construction of acompletely custom device. All other features, including remote services,are also available to the confined individual. It is instructional tonote that in much the same way as the head device is extended to a morecomfortable location for wheelchair and bed access, it could also beextended across a table for meal service orders—although useful inretirement and nursing homes, it also has application in commercialrestaurants.

The head sub-assembly can also be customized to both extend in heightand to tilt backwards in a counter-balanced fashion such that such headdevice is facing upwards at times. In this configuration for example, aDoctor can use the virtual presence robot's head device to take noteswith accurate time and location stamps, or review files while not havingto carry a table from room to room in a hospital. This frees theDoctor's hands to perform procedures and, with appropriate navigation(and ideally, pre-programmed information about the location of eachpatient), speech recognition, and gesture recognition algorithms runningin the head device, the virtual presence robot can move from room toroom following the Doctor and automatically display various files andtake notes (both written, aural, and visual) without the Doctor actuallycoming into physical contact with the virtual presence robot—therebyreducing the chance of physical contamination. The modular nature of thevirtual presence robot enables this type of customization without theneed to design an entirely custom system or new mobile platform. Whenone vertical market, such as the Hospital scenario defined above,justifies the development of a new module like the backwards tiltinghead sub-assembly, application developers from other vertical marketscan leverage ubiquitous web application stores to offer and quicklydeploy new specialized applications, targeted in their area ofexpertise. In this way, the backwards tilting head sub-assembly mightfind applications in vertical markets such as laboratory and clinicaldeployment, industrial monitoring and reporting, and security sweeps.

Going back to the hospital example above, the height of the headsub-assembly can also be customized so that it can closely interfacewith hospital equipment—for example, to periodically, and autonomously,capture a photograph of each IV bag in a hospital ward for automatedvision processing, ensuring an extra check of fluid level and/or timingof replacement.

FIGS. 11 through 14 illustrate two different mid-section modulesdesigned for particular applications in addition to the telepresencefunctionality.

For example, a mid-section sub-assembly can be provided for openingdoors in institutions or homes. Different door levers would requiredifferent relatively low-cost mid-sections or universal manipulator armsand grippers as shown in FIGS. 11 and 12 can be deployed. Since manyhospitals have standardized, easily opened door handles, asingle-purpose actuator requiring only one additional motor is all thatwould typically be required. As budgets permit, deploying a mid-sectionsub-assembly with universal manipulator arms broadens the number oftasks possible. Irrespective of the approach selected for themid-section, special purpose components such as these may be augmented,where fire or other heavy doors are opened, with higher torque motors inthe base sub-assembly having sufficient short-term torque abilities forthese intermittent applications. The omni-wheeled base enables thevirtual presence robot to move with the precise curvature of the door.By adding appropriate autonomous navigation algorithms, the virtualpresence robot can be called via a patient call button to any room in ahospital.

Looking more closely at FIGS. 11 and 12, the modified mid-section module14 a includes opposed robotic arms 37 and 39. These driven arms includepivoting joints (generally corresponding to shoulder, elbow and wristpivoting) and a gripping function (grippers 41) to perform variousfunctions for manipulations of objects about the robot by a remoteoperator or, in the example noted above, to open doors or perform otherautonomous actions.

The modified mid-section 14 b of FIGS. 13 and 14 includes selectivelyclosable cup or beverage holders 43 to assist in delivery of food and/orbeverages to individuals in restricted environments such as hospital orlong term care facilities. This mid-section design can also be used forrestricted access conditions.

Referring back to FIG. 1, the base sub-assembly 18 is illustrated as anomni-wheeled device capable of moving in any direction. The basesub-assembly includes three omni-wheels 20 mounted at 60 degrees to eachother forming a triangle. Those experienced in the art will understandthat there are many types of omni-wheels, including some with doublewheels on each axle or other designs including omni-ball wheels which,for the purposes of the present invention, although some achieve aquieter, less vibration prone movement across hard surfaces, all achievethe desired ability under programmed motion control to effect varyingspeeds in varying directions.

Between each pair of omni-wheels 20 is located at least one, and in thepreferred embodiment two, ultrasonic sensors 38. The ultrasonic sensorsprovide feedback necessary for the embedded motion controller to avoidobstacles. These are preferable to touch-sensitive bumpers due to thepossible inertial consequences of a sudden stop, typified of thoserobots which rely on a physical bump or touch before decelerating. Sincethe base 18 is designed to operate effectively with a variety ofpayloads (a variety of transitional, mid-section, and head subsections,plus the mass produced tablet or other head) carrying out a multitude ofdifferent applications, the motion control capability of the base isfinely tunable to avoid sudden stops or lurches. If transportingliquids, those skilled in the art will appreciate that any increase ordecrease in speed or change in direction conforms to wave-cancellingmotion profiles.

Referring FIG. 4, a perspective view of an embodiment of the basesub-assembly 18 with the transitional section removed illustrates aninternal baffle 40 which can be injection molded in one piece and whichprevents customers from entering the machine compartment below when theyare initially assembling the robot or have disassembled the transitionalsub-assembly from the base sub-assembly to upgrade components. Thebaffle has two twelve volt battery compartments 44 in which batteries 42are inserted. These customer-installed batteries may be easily replacedor upgraded. In the illustrated embodiment, the batteries take the formof common, mass produced, rechargeable tool batteries which aretypically available in economical Nickel-Cadmium and more power-dense,Lithium-Ion types. Through the side of the base sub-assembly is a port54 where the battery connector plate 52 is secured. Note that this port54 is normally covered with a simple assembly holding the ultrasonicsensors, and is not customer removable. Note also that the two batteries42 are either side of the rear wheel, when the virtual presence robot isfacing forward so that in the unexpected event of a sudden forward stop,the weight of the batteries helps to counter-balance the inertia of thehead. When moving backwards, the rear wheel's perpendicular axis makesthe likelihood of a rear fall, even with a more sudden stop, very low.

The base sub-assembly 18 can also include a separate dock station thatallows for recharging of the batteries when the virtual presence robotis docked therewith.

The virtual presence robot preferably communicates with remotetransmitters and receivers that may be provided as a further source ofinformation. This is helpful for multi-level applications or wherecertain areas are not available to the robot. Remote displays andspeakers can also be present. For example, WiFi connection to a furtherlaptop computer or tablet can be used as a remote input and outputstation.

The diversity of hardware configurations and upgrade options enabling acorresponding evolution and growth of applications for the virtualpresence robot targeted at ensuring a long product lifespan is alsoreflected in the system's online components. Each of the virtualpresence robots has a corresponding online profile at a hub which notonly controls remote access and linkage with affiliated display devices,but also links each robot with a series of services, many of which areonly available with a periodic subscription fee.

Directory services enable web visitors to the hub to find publiclyavailable virtual presence robots through a booking engine for groupvirtual experiences like guided virtual field trips and tours whereby anumber of web users collectively and simultaneously experience a virtualpresence through one virtual presence.

Other bookings through the directory enable web users to reserve avirtual presence robot for their exclusive use for a period of time.Using NASA as an example, in this case web users would check thedirectory for museums and select NASA. NASA, like other large touristexhibits, would offer a number of virtual presence robots for use duringthe day when the facilities are open on a restricted basis, according tohow busy the facility is at a given time of day (since these would be‘mingling’ with other real visitors to the complex) and in the eveningafter normal visiting hours have ended, might reserve a few hours eachnight for the exclusive use of virtual presence tourists. Peak timesduring the day, for example, might be reserved only for teachers who areconnecting with their classes for virtual field trips. Booking timeswould be enforced by the virtual presence robot automatically returningto a central point at the end of any booked time slot so that the nextuser in line could have timely access.

The directory would also feature shopping venues. For example, there area great many painting, furniture, and antique galleries who find itextremely time consuming to list each product for sale and providenumerous views. In the shopping directory, web visitors can see a listof immediately available shopping experiences where virtual presencerobots are currently available or could schedule a visit to a selectedgallery based on virtual presence robot availability. The configurationof the virtual presence robot will vary by store, but typically thesewould be configured with BLUETOOTH™ communications so that a salesclerkcan be called and can even answer via an earbud when more productinformation is desired without the user of a virtual presence robotactually crying out across the store.

The directory would also feature forums for hotels, restaurants, casinosand other public places where virtual presence experiences are offered.In every case, the virtual presence robot can be limited by the host tomove in only selected areas and can also be limited such that it cannotpoint its camera in certain directions. This would also enable, forexample, tours of portions of NASA typically off-limits to the generalpublic. The virtual presence robots enable a massively expanded arenafor personal exploration without the threat of vandalism or acts ofterror. The base sub-assembly can also be upgraded with larger tires andmotors for outdoor and rugged terrain and the transition sub-assemblycan be upgraded for automatic stabilization.

Hospitals could also offer a set of virtual presence robots wherefriends and families of patients could book a virtual presence visit.Versions of the virtual presence robots in this scenario would includeautonomous navigation capability, so that the robot would be at thebedside in advance of communications and might also requireacknowledgement from the patient before accepting the communicationlink.

Private, login access only portions of the directory or virtual presencehub would enable pre-authorised users to see the real-time status of theprivate virtual presence robot (including availability, battery level,and tablet or other head device connection status that is physical orwirelessly connected to the robot) and initiate connections. Forexample, to monitor a remote property during absence, a user can loginto their virtual presence robot on site and tour their property. Theserobots can be packed away, out of sight when physically on the property,so the annoyance and invasion of personal privacy felt with fixedsecurity cameras is not an issue with virtual presence robots. Whenlogged into any virtual presence robot, commands can be sent by the userto program the virtual presence robot to initiate a connection back tothe user at certain times, or if certain events occur. An example mightbe a loud noise, sudden drop or rise in temperature, or unexpectedmotion events (curtains moving, etc.).

Real estate agents could also offer regulated logins to prospectivebuyers. In the case of unoccupied homes or condominiums, 24 hour accessto virtual presence robots on each floor or in different areas of thefacility could be offered. In the case of occupied private dwellings,limited ‘virtual open house’ hours could be offered. Questions during avirtual tour can be set to be directed to the cell phone or other deviceof the agent's choosing. The outdoor capable version of the virtualpresence robot would be necessary for recreation property viewing.

Going back to the private login directory, friends and family memberscan get different levels of login to the virtual presence robot. Onelevel requires acknowledgement from someone in the presence of thevirtual presence robot before communications can be established. Thisprevents unexpected visits. Another level might permit login and theestablishment of only audio communications without acknowledgement. Thehighest level access permits login without acknowledgement and would beparticularly vital for people who have ill or mobility impaired friendsand family who wish to have periodic check-ins. Through the centraldirectory and hub, linkages from all social media websites can beestablished, so the availability of a virtual presence robot at a friendor contact's current location would be known, no matter which socialmedia platform was being used. Virtual presence robots may also beassigned a standard POTS telephone number so that they may be ‘called’from universal telephones and, depending on the caller-ID of theinitiator of the call, appropriate login access will be granted.

Extension of the virtual presence robot's processing and informationresources through online services enables further customization of theplatform. For example, an application running on the third party headdevice could, in conjunction with an online service, monitor and learnthe daily pattern of an elderly family member. When events occurredoutside of the pattern, say breakfast had not occurred by 11 am, theautomated portion of the service might telephone the elderly person andif no answer was received, a programmed set of events would thenoccur—likely beginning with a call to a pre-designated family member whocould indicate via touch-tone or spoken commands, whether the eventshould be ignored or further escalated.

In any location where a virtual presence robot is located, and giventhat it has the appropriate autonomous navigation and voice recognitionapplications installed, it can be hailed through smartphones or otherWiFi devices or simply called by a large shout, if so programmed. Oncein the immediate vicinity of a person, it can be verbally interactedwith to establish a communication channel with a friend or associate'svirtual presence robot or to perform information gathering (ie: What isthe weather forecast? What movies are playing on TV tonight? When is thenext AMERICAN IDOL™ on TV?) and provide services (ie: “Please turn onthe TV and tune to AMERICAN IDOL™ when it's next on.” “Order some SWISSCHALET™ Chicken—two half chicken dinners with fries and a bottle ofwater.” “Read me today's top news stories.” “Read me the latest WilliamShatner book beginning where I last left off.” “Call my brother.”). Ifthe third party head device does not have enough processing power toperform the voice recognition and formulate the web queries orsuccessfully navigate from room to room, the virtual presence robot canstream the captured audio and video to web services to perform theoperation seamlessly from the user's perspective. This enables thevirtual presence robot to potentially leverage massive remote computingpower to truly simulate dialog with another person and even receive adetailed simulated ‘face’ on the display with lips moving insynchronization with the audio arriving from the service. Web servicesfrom a broad variety of suppliers can be integrated—for example, onlineebookstores for reading material, and common VOIP phone servicesoffering VOIP connections to POTS lines and video calling could offerapplications which would seamlessly integrate their large user-basesinto the virtual presence robot.

With the appropriate mid-section or head sub-assembly, games could alsobe played by projecting a game board onto a table. True virtual presenceis achieved when the communications with such remote services are asfluid as communicating with another human being. Users do not have to goto a special room in the house or sit in front of a computer or holdtheir smartphones, they can simply call out for the virtual presencerobot to come to them and then link with Cloud computing services or toreach out to friends.

Additional details of the base module 18 are shown in FIGS. 4, 5 and 6.The internal baffle 40 as shown is removable as illustrated in FIG. 5 toexpose four USB ports 46, one DC power connector port 48 and onemulti-pin interface connector port 50. Dedicated wires from the USBports 46 with standard connectors on each end can be easily run withinthe upper modules of the virtual presence robot during customer assemblyto USB components such as the camera capturing the 360 degree fields inselected mid-sections, the speaker/microphone array and possible RGBDdevices like the Microsoft KINECT in selected head assemblies (see FIGS.15 and 16 and the modified head module 12 a) and to enable charging andoptional communication with the tablet at the docking port. The DC powerconnector port 48 provides power for devices either controlled throughthe I2C bus or those connected via USB but exceeding the standard USBpower specification—for example, certain speaker setups orprojector-enabled mid-sections or a Microsoft KINECT which requiresapproximately 15 watts of power necessitating a USB connection coupledwith an additional power input to correctly feed the modified USBconnector on the Microsoft KINECT. The DC power connector also providespower for motors including the head tilt mechanism or pill dispenser,and may be daisy chained from motor to motor, for example, the mechanismwith three degrees of freedom (employing three motors) to extend headsupport beyond the base sub-assembly to bring the head device withineasy reach of users sitting in wheelchairs and resting in beds. Lastly,the multi-pin interface connector port 50 provides an I2C bus, controlpower, and both analog and digital inputs and outputs which are alsodesigned to support parallel data capture devices.

The I2C bus and control power provided by the multi-pin interfaceconnector port 50 may be run throughout the virtual presence robot asnecessary to daisy-chain low bandwidth peripherals such as the statuslights and button array, laser pointer, control for the head tilt, andcontrols/sensors for a multitude of possible options including thepreviously detailed drug dispenser, cup holder, head extensionmechanism, and various automated latch mechanisms. Small,power-efficient microprocessors with embedded flash memory includinganalog to digital converters and I2C bus support may be hung off the busat any point and at nominal cost to operate numerous mechanisms asrequired for various vertical markets and/or different applications.

The housing 54 as shown in FIG. 5 is preferably injection molded withminimal post machining and is one continuous piece with the motormounting plate 60. Small ventilation holes behind each wheel oppositethe motor mounting plate enable air to flow into the base withoutexposing electrical hazards and air is expelled through a mesh at thebottom rear of the unit with the help of a small DC fan.

For illustration purposes, one battery 42 remains in its batteryconnector plate 52 while the other is missing. Wires running from eachbattery connector plate 52 to the main circuit card containing theembedded microcontroller 64 carry current from the batteries to the maincircuit card during system operation. During charging, current isreversed. As will be familiar to those experienced in the art, in asimilar embodiment it is possible to eliminate these sets of wires byenlarging and repositioning the main circuit card so that it spans thearea under each battery mounting position, enabling the batteryconnectors to be located directly on the main circuit card andeliminating the separate battery connector plates. Furthermore, suchexpanded card could also reach past the left or right battery positionto mate with an induction or contact-based linkage with an externaldocking station for electrical supply during charging. A variety ofdocking options exist which are commonly known to those skilled in thetrade.

Looking again at the base sub-assembly housing 54 in FIG. 5, thepreferred embodiment employs brushless DC motors 62 mounted to the motormounting plate 60 such that the notched pulley on the motor shaftextends through the mounting plate 60 in line with the notched drivepulley 56 locked to the wheel axle 58. Depending upon the availableheight of the base housing 54 which impacts the aesthetics of thevirtual presence robot and must contain the notched drive pulley 56, thetorque from the motor may be increased approximately 4-5 times throughthe ratio of the size of the notched motor pulley and the larger notcheddrive pulley 56 on the wheel axle. In cases where a higher gear ratio isnecessary, such as with the use of a smaller motor or to traverse ruggedterrain, worm gear or planetary gear reductions are also possible withinthe same physical lower base housing.

Although DC gear motors are used in other robots to directly drivewheels at a lower cost and without the controller complexity and needfor pulleys and belts, the advantage of the gearless, brushless DCmotors connected via belt reduction systems in the preferred embodimentof the present invention are multi-fold: (a) this belt reduction systemis much quieter than gear reducers, more closely duplicating relativelysilent human motion, (b) the motors spin at much lower speeds,significantly increasing longevity, (c) the motors do not use brusheswhich, necessary for traditional DC motors, wear out withuse—particularly high speed use where a high gear reduction is used toachieve desired torque from a smaller motor, (d) the drive-train is notsubject to damage if it is back-driven through an external force whereasmany of the inexpensive high ratio gear reducers fail if back-driven andby resisting such forces, make it difficult to manually push or relocatethe robot in the event of control or electrical failure, (e) brushlessDC motors are typically sensor driven, meaning that they containhall-effect or other position sensors which when coupled with advancedmotion controllers, report on the rotation of the motor—and their motorleads (typically for smaller brushless motors) package such sensorcables in a bundle with coil cables, negating the need for a separatelyinstalled shaft sensor and two sets of connectors per motor, and (f)using advanced, current-sensing motor controller design, it is possibleto detect unexpected motor load changes and virtually instantaneouslyshut down the motor.

The preferred embodiment of the present invention facilitates quickinstallation of the motion components illustrated in FIG. 6, within thebase sub-assembly housing 54 illustrated in FIG. 5. A notchedhigh-strength polymer shaft 58 with a cap at one end is inserted into anotched center of omni-wheel 20 and then through a planar polymer flangebearing which is pressed through a hole in the sub-assembly housing 54then through a polymer washer and through a notched center of thetoothed pulley 56 and belt 68 (as shown in FIG. 6) and through a secondpolymer washer then through a hole in the motor mounting plate. At thefar end of the shaft, a second polymer flange planar bearing is pressedinto the hole in the motor mounting plate and the end of the shaft 58projecting through the flange is locked in place by a split-ring. Themotor 62 with toothed pulley is pushed through an over-sized hole in themotor mounting plate and through the belt 68. The motor is then moved ina direction away from pulley 56, to remove slack from belt 68 and thensecured with four screws to the motor mounting plate. A small about ofbacklash, due to remaining slack in belt 68 is not critical to theoverall motion and trajectory of the virtual presence robot sincetraction variations and general properties of a 3 wheeled omni-wheelsetup do not guarantee precise motion. Furthermore, because the width oftoothed pulley 56 and the motor pulley are only slightly thinner thanthe space between the exterior base housing 54 and the motor mountingplate 60, should the belt jump or slip, there is no possibility of itjumping entirely off the pulleys. The motor mounting plate 60 alsoshields ingress to the electronics from any projections into the ventholes behind the wheels opposite the motor mounting plates and alsoshields any floppy wires from becoming entangled in the belts.

The main circuit card 64 containing the embedded micro-controller, inaddition to the wires running to the battery connectors, wires run fromthis card to each motor (typically hall effect sensor lines and coillines unless each motor has an embedded controller, and then in thiscase, power, and I2C bus lines to each motor), to the ultrasonic sensorssurrounding the base, to a small ventilation fan facing downward througha grill at the back bottom of the robot, and to the baffle connecting toeach of the USB ports 46, the power connector 48, and the multi-pininterface connector 50.

The main circuit card 64 has a number of embedded systems necessary tosupport the third party head device and various head, mid-section,transition, and base sub-assemblies via either USB or I2C bus interfacesor wireless communications (typically BLUETOOTH™ or WiFi). Data fromthese devices and sensors which is not wirelessly passed to the headdevice, along with motor control and diagnostic information, isprocessed through the main circuit card and presented as API'saccessible through either the USB or wireless link between the embeddedsystem and the head device. Although the head device is typicallyconnected to the USB for charging purposes while docked, all data willbe passed via wireless protocol as this enables the head device(typically a tablet computer) to continue to communicate with thevirtual presence robot, even while removed from its head sub-assembly.In some cases, where wireless range is too limiting or deemed to beunsecure and a continual connection with the virtual presence robot isvital, the embedded system can also implement Cellular or proprietaryprotocols to ensure that the head device is connected as broadly aspossible. For example, if medical or security monitoring devices arelinked with a special transition sub-assembly containing logging andalert processing, the transmission of such alerts to the head device,which in turn makes connections via the Internet only after furtheranalysis of the nature of the alert, is likely to be seen as a criticalcommunications link and would ideally follow two or more possiblewireless routes in the event of an emergency alert.

Except in the case where each motor has its own embedded controller, oneof the key processors on the main circuit card is a FPGA whichsimultaneously monitors the hall-effect sensors in each brushless motor62 and varies the pulses to each of three coil phases in each of themotors to precisely, and simultaneously, control the speed of each motorto effect the desired overall motion of the virtual presence robot. Ifeach motor has its own embedded controller, then similar speed controlsignals are passed to individual motor controllers daisy chained viaI2C. Trajectory adjustments to accommodate varying wheel slippage andtransition between floor surfaces and over surface inconsistencies andto accommodate navigation demands are made from vision, ultrasonic andother available off-board sensors (depending upon which mid-section andhead functionality is available), as well as an on-board gyro andaccelerometer, and are made 20 times a second. Smooth, life-like motionis possible with this combination of sensors. In unexpected emergencysituations, for example, if a dog were to jump up on and begin to topplethe virtual presence robot, the motors can accept momentary high-currentpulses outside their normal operating range for extremely quick motionrecoveries—in this example, to move the base as quickly as possible inthe direction of the possible head topple.

The main circuit card 64 also includes a DSP (or other image processor)which is typically used to reconstruct the 360 degree scene (in either2D or 3D) from the camera video of the reflections in the dome (ordomes). In conjunction with ultrasonic sensor data from the base, theDSP also calculates potential collisions and other hazards (such as thetop of a flight of stairs, for example) based on ideal trajectoryinformation from the third party head device. Irrespective of motioncommands from this head device which may also be performing moreadvanced 3D processing for navigation should it access the 360 degreecamera stream via API, the embedded system in the base will not let thevirtual presence robot collide with objects or fall off steps andledges.

Certain devices connected via USB may directly communicate to the headdevice, assuming the head device is connected in the head sub-assemblywith USB for both power and data, without any intervention fromprocessing in the base. This facilitates the inclusion of proprietarythird party devices controlled by third party software executing onstandard head devices to be added to the robot, where the head devicereceives USB data, without specific need for software updates to theembedded systems running on the main circuit card. By enabling thirdparties to attach such devices and thus provide augmented sub-assemblieswithout the need for embedded system updates, small vertical markets maybe cost-effectively addressed by such third parties who can leveragemanufacturing volumes of standard sub-assemblies and the overall robotplatform.

Where the head device does not receive data via USB (i.e. it isconnected to USB in the head sub-assembly for power only), or is removedfrom the head sub-assembly yet still requires a communications channel,or the USB device is accessed through a generic API provided by embeddedcode, processing in the base sub-assembly provides documented API accessvia a wireless connection such as BLUETOOTH™ or WiFi. Note that in allcases, devices connected via I2C will have documented API access overeither USB or a wireless connection to the head device, depending uponthe configuration of the head device. Standardized commands for motorcontrol (ie: for use as a cup lock mechanism), laser pointers,solenoids, status lights, and other I2C devices will also enable thirdparty devices to integrate in the virtual presence robot and becontrolled by the head device or remote web services without the needfor custom code running on embedded systems in the base.

For further clarity of the head options, examine two examples in FIGS. 7and 8 and the further head 12 a of FIGS. 15 and 16. In FIG. 7, a thirdparty tablet 10 can be seen sliding towards the left between the guidetabs of the tilting mounting plate 22 to mate with connectors in thehead sub-assembly 12 which also includes a speaker, microphones, atilting apparatus, and numerous status lights and button. In FIG. 8, anintegrated head sub-assembly 72 containing a laser pointer apparatus 76,a large integrated display 74, as well as processing power, and wirelessconnectivity adds only one third party device—in this case, a SonyPlaystation Eye™ camera and microphone array 70 mounted on a tiltmechanism. Both head sub-assemblies connect to the same mid-section,transition, and base sub-assemblies (the illustration in FIG. 8 is asmaller scale than that of FIG. 7).

FIGS. 7 and 8 however, target different markets. The head sub-assemblyin FIG. 7 is fairly inexpensive, as the true cost and much of thefunctionality of the overall head is in the third party tablet devicewhich offers the cost/performance benefits of truly massive productionvolumes. This device coupled with the head sub-assembly is likely to beused in an office or light commercial environment where the backgroundnoise is too high, in many cases, for the very small built-in speakerwithin the third party tablet—thus the inclusion of a more powerfulspeaker and microphone array in the head sub-assembly. This head alsotilts, again useful for an office environment where looking down at adesk or machine is useful—as is looking up to a whiteboard during ameeting. No additional camera is necessary, as the tablet includes bothforward and rear facing video cameras running at 30 frames per secondwhich is good for slow movement, typified of motion in tight spaces. Themore extensive light array is necessary since the base may be lefteither in its charging station or elsewhere with the tablet removed. Inorder to show battery level and whether or not charging is occurringwhen the tablet is removed, multiple lights are thus required. Further,an indicator light is used to show when the tablet has properly dockedand if the ‘do not disturb’ function has been activated. Two buttons,one for the ‘do-not-disturb” function and another for the ‘sleep’function, are provided.

The head sub-assembly in FIG. 8 with the Playstation Eye™ can be used ina different environment. It has a larger touchscreen display, better formore distant viewing, yet doesn't include a speaker or tilt capabilityon the display (the weight of the larger display means that a forwardtilt would significantly increase the chances of a forward fall in theevent of an unexpected deceleration and thus tilt cannot be used withlarger displays unless a counter-balanced head sub-assembly is used).When equipped with a Playstation Eye™ combining a high-speed videocamera and microphone array on a tilting mount, this configuration makesan ideal virtual field trip presence robot or virtual shopping robot.For example, in the virtual field trip scenario, a guide walkingthrough, say NASA or the Vatican, would carry a Playstation Move™ motioncontroller wand. The application would use sensor data from thehigh-speed Playstation Eye™ camera and Move™ controller to ‘follow’ theguide and look to the items that the guide gestures to throughout thetour. The high-speed camera enables the guide to move fairly quicklythrough the tour where necessary, as images taken from the movingvirtual presence robot will not be too blurry. The array microphonereduces background noise, yet still conveys the ambiance whileBLUETOOTH™ (or other wireless technology) headphones with integratedmicrophone enables the guide to hear and answer questions during thetour which can be autonomously sequenced by the online serviceconnecting remote tourists to the tour.

At various points in the tour, the guide can ask visual questions whichthe entire group of virtual tourists can answer by clicking or touchingobjects seen on their screens—the laser pointer then projects pointclouds on the actual objects or areas indicated by the tourists, with adot representing each tourist's selection. Buttons on the PlaystationMove™ could also be used to indicate the beginning of a question whichwould instruct the system to use voice recognition to place a textualversion of the question on each tourist's screen along with multiplechoice or written answer replies which would be compiled in real timeand communicated to the tour guide via the virtual presence robot'sdisplay or using computer generated speech, spoken to the guide overtheir wireless headphones. The guide could also hold up small writtenquestion and multiple choice answer cards which the virtual presencerobot would recognize and immediately broadcast to each virtualtourist's screen for answer by touching the correct answer or in thecase of a written answers response, an input box. Alternatively, verbalreplies from the virtual tourists can also be processed through speechrecognition techniques before being compiled and in this way, it wouldbe possible for the tour guide to ask a verbal question to the touriststhrough either the array microphone or through a BLUETOOTH™ ear bud orheadset microphone and then hear, or see displayed, the compiled resultsvirtually immediately after asking the question.

An alternative to the Playstation Eye™ would be a Microsoft KINECT XBOXwhich can track hand, arm, and leg movements without the need for awand, albeit through a slower camera. The Microsoft KINECT device itselfincorporates a tilting camera mount, so the head sub-assembly 74 wouldbe significantly less costly for the Microsoft KINECT than for thePlaystation Eye™ as illustrated in FIGS. 15 and 16. In thisconfiguration, targeted at security applications for Microsoft KINECTXBOX gamers, beyond sight and sound (RGBD camera and microphone array inthe Microsoft KINECT, and speaker in the head sub-assembly), the headsub-assembly as shown in FIGS. 15 and 16 has only a button/status lightinterface. In this extremely low-cost configuration, no processing otherthan the embedded circuit card in the base sub-assembly is availablelocally. Although the head sub-assembly could be augmented with alow-latency cellular data connection (ideally LTE or 4G speeds orfaster) for Cloud computing services, typically, a WiFi connection fromthe embedded circuit card would link via a low-latency connection toremote processing as the robot cannot provide advanced navigationcapabilities using solely the embedded circuit card. For securityapplications, a direct connection with cellular data is preferred.

As in the security application example above, in the XBOX gamerscenario, the head sub-assembly is the same minimal configuration.However, the XBOX now becomes the remote processor—visitors of gamerslog into the virtual presence robot and are able to move about in theroom. The robot makes a WiFi connection to the XBOX (or vice-versa),where the visitor's face will be caused to appear (in a small window) onthe TV screen. Thus, the gamer can continue the game while their visitorcan watch and be entertained—through the robot, they can choose to facethe TV screen or watch the gamer at the controls. Of course, headsub-assemblies offering richer local functionality incorporating theMicrosoft KINECT are also likely, including those with dedicated LCDpanels or combining a Microsoft KINECT with a tablet—but for a gamer ona tight budget, the head assembly with solely the Microsoft KINECToffers a unique way that friends who are not playing the game, can joinin the fun.

FIG. 9 illustrates a basic connection initiation screen from which aconnection to a specific virtual presence robot is established based oneither the owners name 80 or alternatively, the virtual presence robot'sname. This application integrates with the video and audio capabilitiesof the remote host platform device including desktop, laptop, and othertablet computers as well as smartphones, web-enabled game consoles andtelevisions. Details of the connection to be established such as thevisual clue 84, time of last successful connection 82 and access to thelast connection 92, connection history 90 and a full contact list 88 isalso provided.

FIG. 10 illustrates an embodiment of some of the controls a remoteconnected user of the virtual presence robot is able to access from asmartphone or tablet. A similar experience is possible from any numberof remote host platforms from Web-connected televisions (where arrowkeys on the TV remote, or wireless pointer, would be used for buttonselection and navigation targeting) through desktop PC's (where themouse would be used instead of touchscreen). The main view is thelargest portion of the screen and is filled with the main camerareal-time video view 94 broadcast from the virtual presence robot.Normally in 2D, if the virtual presence robot is equipped with a 3D maincamera, and assuming the remote user has access to a 3D display ordisplay/glasses combination or 3D goggles, this area will be received in3D at a frame rate dynamically determined by the Internet connectionspeed. A sub-frame video 96 shows what the remote user is transmittingfor display on the head screen of the virtual presence robot. A panorama360 degree view of the immediate surroundings of the base of the virtualpresence robot is shown in a wide window 98 (in either 2D or 3D,depending on the capabilities of the virtual presence robot and theuser's equipment) so that in the event that manual navigation mode isselected by touching 102 and then interacting with a virtual joystickcomplete with spin-head, the remote user can see what might be blockingmovement in a given direction. The video frame rate transmitted from thevirtual presence robot for this panorama 98 is significantly less thanin the main view unless the Internet channel is sufficient to supportdual real-time streams. The panorama 98 can also be hidden 110 whichreduces bandwidth demands. In the illustration of FIG. 10, the panorama98 is shown as a continuous panorama which, as previously noted, musthave some obstructed areas generated based on earlier frames. Theconnection may be terminated by touching the ‘end’ button 112.

For some host platforms, the virtual joystick is extended by swipe andother common touch screen gestures which are then transmitted to thevirtual presence robot. For example, a double tap anywhere on the screenwill cause that area of the image to immediately enlarge via imagezooming while the virtual presence robot then turns to face and movestowards, and thus present a clarified view a few moments later.

In addition to the virtual onscreen joystick control in manual mode,interface capabilities of the remote host platform are integrated intothe control system. For example, for the Apple iPhone™ and similarlyequipped smartphones and tablets, a toggle option labeled ‘virtualtracking’ appears which, when enabled, links physical movement of theiPhone™ to be duplicated by the remote virtual presence robot wheremechanically possible and without collisions or topples from stairs orwindows. For example, if the user pushes the iPhone™ away from theirbody by extending their arms out straight or walks forward with theiPhone™, the virtual presence robot will move forward, in the directionof the iPhone™ motion. If the user then spins left or right, the virtualpresence robot will rotate similarly. The screen on the iPhone™ willshow the resulting motion which, depending on network lag and the speedof the motion, will lag behind the users motion. Quick shakes of theiPhone™ in one direction will move the virtual presence robot in thatdirection, as will tilts of the iPhone™—if equipped with a head tiltsub-assembly, the head will correspondingly tilt up or down to match themotion of the iPhone™. Similar experiences can be achieved with otherremote host platforms.

Target navigation mode may be selected by touching 100, and in thismode, the user can then tap anywhere in either field of view 94 or 98and the virtual presence robot will autonomously move and/or rotate to aphysical location as close as possible and directly facing the selectedtarget. A longer touch and then slide motion will drag out a rectangleover the field of view which will become a broader target location forthe virtual presence robot. When in target mode if the “virtualtracking” toggle is enabled, during autonomous navigation to a selectedpoint, the user can move and twist and turn the iPhone™ to adjust thepath and any taps of the screen will change the target while the virtualpresence robot is in motion. The robot always attempts to smooth motionso that resulting video streamed to the remote user will be as smooth aspossible.

Other options available from the remote host platform include theability to increase or decrease the remote speaker volume 104—handy whencalling out to someone through the virtual presence robot. Where deviceshave both forward and rear cameras, they may be swapped 106 at both theremote host platform and on the virtual presence robot. Photos andvideos that may be accessed from the remote host platform may also beshared 108 with the virtual presence robot and will be displayed on thatrobot's display, or where such robot has one or more registeredaffiliated display devices (typically web-connected TVs) redirected toone or more of such device while keeping the remote user's face on thevirtual presence robot's screen. The virtual presence robot can alsoduplicate, at any time, its display content onto one or more registeredaffiliated display devices. This capability to include affiliateddisplays within the virtual communication session is very useful whensharing photos among friends and family, but is also highly desirablefor distance learning whereby a guest lecturer or other educator canvisit a remote classroom via the virtual presence robot and have theirface on the screen of the robot while simultaneously sending a secondstream of still images or video (including PowerPoint or other slides)which the virtual presence robot can re-direct to a number of affiliateddisplay devices.

To fully exploit the virtual presence capabilities of the modularsystem, upgraded remote host platforms and different head units may beused. This is accommodated in the present modular design. For example,3D goggles employing an accelerometer and gyro designed to enable thewearer to move their head to freely, and seamlessly change the viewreceived from the virtual presence robot may be supported by upgradingthe robot's head unit to capture, via two or more cameras, a far greaterfield of view than actually being viewed in the 3D goggles at any onemoment. Although bandwidth intensive, by transmitting this broader 3Dview, the remote host platform can reconstruct an immersive view whichwill exhibit no lag during physical head movements within a givenenvelope.

The virtual presence robot design anticipates advancements in each ofthe fields of 3D display technologies, high speed global datatransmission, reductions in network latencies, and camera technologies.Today, a trade-off exists between the amount of visual data streamedfrom the virtual presence robot, network latency, and the range ofseamless motion that the remote 3D goggles can support. A head unitwhich transmits a field of view 50% larger than displayed in the remote3D goggles would enable such remote user to turn their head in anydirection in a fairly deliberate, relatively slow manner and experiencea seamless, smooth view as the virtual presence robot, with networkdelay, autonomously moves to keep streaming video centered ahead of theanticipated desired view.

The present design also supports special 3D virtual presenceexperiences. For example, by equipping the virtual presence robot headunit with an array of cameras and microphones capturing a full 360degree, 3D, high resolution view and multi-directional sound, coupledwith high-speed streaming data services, the virtual presence robotwould enable any number of remote wearers of 3D goggles to stand or sitand rotate at any speed and in any direction as if they were truly atthe spot where the robot was located. Although this would require that aconsiderable amount of data be transmitted, cable, satellite or fiberInternet providers, using multicast protocols, could cost-effectivelytransmit such streams in real time enabling hundreds of thousands ofsports fans, for example, to all share the same front row seat.

Likewise, for educational purposes, students wearing 3D goggles sittingtogether in a classroom or lecture hall could look in any direction.Seating could be modified such that each student's chair is free torotate 360 degrees without students colliding. Taking the virtualpresence robot to venues such as dairy farms, maple sugar bushes,electrical generating stations, ship navigation decks, hospitaloperating rooms, submarines, building demolitions, and hundreds of othervenues enables a new level of experience which would be impossible tophysically execute with a class of students for safety or transportationlogistics. Where real-time streaming is not possible due to networkbandwidth limitations, the head device of the virtual presence robotcould act as a recorder for later transmission.

Before describing the virtual presence robot 2 of FIGS. 17, 18 and 19 itis noted that the head module 12 thereof is shown as two pieces 13 and15 having a pivot pin 17. Component 15 is shown with a connectedcomputer tablet 19. The head module 12 is shown in this non-connectedview to help illustrate a simple pivoting arrangement for mounting thecomputer tablet 19. In the preferred embodiment the head module is anassembled structure to the consumer who merely docks a third partytablet or other device to the head module 12. FIG. 7 shows the preferredslide mount of the third party device.

FIGS. 17, 18 and 19 illustrate the connections between the series ofmodules and the effective conversion from the generally triangular basemodule 18 and the triangular transition module 16 that then connects atan upper edge with a generally planar mid-section module 14 and the headmodule 12.

The base module 18 and the transition module 16 include a series ofvertical projecting members and appropriate locking slots provided atthe lower edge of the transition module 16 to effectively secure thetransition module to the base module. A series of pin connectors areshown to effectively connect the mid-section module 14 to the transitionmodule 16 and similar pin connections connect the head module to themid-section module. FIGS. 17, 18 and 19 also illustrate the positioningof the dome shaped reflector within the port 35 of the mid-section andthe extent that the dome reflector extends front and back of themid-section module 14.

The base module 18 is preferably of a width at least 2 times greaterthan the maximum width of the mid-section module 14. The head attachmentmodule is preferably of a width less than the width of the mid-sectionmodule. The transition segment tapers inwardly to form the transitionfrom the triangular shape of the base module to the generallyrectangular shape of the mid-section module. The mid-section module hasfront and back faces preferably at least 4 times the width of side facesthereof. The base module is typically of a weight at least five timesthe weight of modules supported thereabove.

FIG. 20 illustrates the communication paths between a remote PC ortablet via the Internet WiFi, 3G or 4G network to the robot. It alsoillustrates the communication within the robotic structure to thevarious motor drives, the various ultrasonic sensors, the panoramiccamera and processing provided within the robot. Additional remotesensors and receivers are not shown but can be used via WiFi or Cellulardata networks, as can the remote processing and service hubcapabilities.

FIG. 21 is an electrical schematic showing the connection of the variouscomponents within the robotic structure.

The present application describes a series of applications of thetelepresence robot and includes applications where the communicationbetween the robot and a person can be short range applications. One suchapplication is with respect to providing retail assistance to potentialcustomers initially by the robot with the capability to contact a storeclerk if additional assistance is required by customer. In thisparticular example, the robot is contacting the sales clerk or furtherresource where the initiating individual's original communication iswith the robot. This approach is also useful for other applications. Forexample, a person could contact a robot provided in their house using acell phone to instruct the robot to undertake certain actions. Theseactions may be as simple as turning on lights within the house to otheractions including activating automatic locks or being in a suitableposition to confirm appropriate authorization through recognition oflicense plates, recognition of facial characteristics of the owner, etc.

It has been previously described that the robotic structure can alsoreceive and transmit signals from remote equipment that may bedistributed throughout a house or hospital. In this way there may beareas of premises that are not directly accessible by the robot howeverthese remote stations can provide feedback or monitoring of theserestricted areas. This may be particularly appropriate for seniorsliving on their own where a remote station is provided on the secondfloor of a house. In this way even though the robot may be confined tothe first floor of the premises, the robot can provide assistance to anelderly person by monitoring the second floor space. In case of anemergency, the robot can determine an emergency condition and initiateappropriate communication to outside authorized personnel.

The various applications of the robot have included a number ofsensors/communication links that provide knowledge and directly affectthe functions of the robot. It is also possible that other links canprovide information to the robot and are not in the form of atraditional sensor. Such additional information can come from outsidecommunications (for example the Internet with respect to weather,television listings and other structured information, but alsocustomized sources such as monitors for individuals, medical personneletc. could all provide information that would affect the operation ofthe robot.

One of the features of the present design is the ability of the mobileplatform to interface or cooperate with third party devices or systemsand related online distribution infrastructure to be compatible withcustomized applications and to effectively utilize existing technologyand leverage the automated distribution capability of related onlineapplication stores. Customized software can be developed with respect tothese customized systems and devices to perform a host of diversefunctions through standardized API's while maintaining compatibilitywith third party hardware and application stores and the generalstructure of the robotic platform.

The robotic structure as disclosed can provide effective monitoring andcommunication with actual events occurring at a remote location.Although this is a desirable function there are situations where suchcommunication is not appropriate. It is recognized that there may beareas within a premises that are identified as being “non-transmitzones” and such zones can be automatically recognized by the roboticstructure. It is also desirable to be able to manually set the robot ina “non-transmit mode” by a worker or authorized individual. This hasparticular application in hospital and medical care facilities and mayalso have applications in certain facilities where high security isrequired. The system is such that it can receive overriding or priorityinstructions that such a telecommunication function is temporarilyinterrupted. Such black out requests can be automatic or manuallyactivated and can be temporary or specific to a certain area orspecified time. As can be appreciated, any of these aspects can beprogrammed and/or recognized as such a condition to provide theinterruption. The robotic design can also effectively transmitinformation indicating that the local “non-transmit” condition has beenundertaken for a particular reason or will be experienced for a certaintime period.

In many applications, there will be situations where privacy provisionsover-ride the telecommunication function and as such the roboticstructure is designed to either automatically or manually enter anon-transmit mode.

Although various preferred embodiments of the present invention havebeen described herein in detail, it will be appreciated by those skilledin the art that variations may be made thereto without departing fromthe appended claims.

The embodiments of the invention in which an exclusive property orprivilege is claimed are defined as follows:
 1. A mobile robotcomprising: a body; a support base supporting said body and having aplurality of omni-directional wheels driven to move the mobile robotacross a surface; a computational unit with wireless networkingcapabilities, the computational unit including directory services hubfor enabling virtual interaction with the mobile robot.
 2. A mobilerobot as claimed in claim 1 wherein said directory services enablesusers on the wireless network to login to a mobile robot.
 3. A mobilerobot as claimed in claim 2 wherein said directory services furthercomprises a database for storing information associated withpre-approved users.
 4. A mobile robot as claimed in claim 3 wherein thepre-approved users can see the real-time status of the mobile robot. 5.A mobile robot as claimed in claim 4 wherein the real-time statusincludes any one or more of the following robot properties:availability, battery level, and connection status.
 6. a mobile robot asclaimed in claim 4 wherein the access of the pre-approved users can beset to allow the pre-approved user to access the mobile robot only withacknowledgement from a user present with the robot.
 7. A mobile robot asclaimed in claim 4 wherein the access of the pre-approved users can beset to allow the pre-approved user to access to audio communicationswithout acknowledgement from a user present with the robot.
 8. A mobilerobot as claimed in claim 4 wherein the access of the pre-approved userscan be set to allow the pre-approved user to access the mobile robotwithout acknowledgement from a user present with the robot.
 9. A mobilerobot as claimed in claim 2 wherein the directory services hub includesa booking system for reserving a time to access a particular mobilerobot.
 10. A mobile robot as claimed in claim 9 wherein said robot isconfigured with software to navigate a known space.
 11. A mobile robotas claimed in claim 10 wherein said robot is used to provide virtualguided tours.
 12. A mobile robot as claimed in claim 11 wherein themobile robot is controlled remotely by the user to view a particularspace in which the robot is located.
 13. a mobile robot as claimed inclaimed in claim 12 wherein said mobile robot is configured to limitvirtual access and virtual viewing to pre-defined geographic locations.14. A mobile robot as claimed in claim 1 wherein the mobile robot isconfigured with autonomous navigation capabilities.